Toshiba BRIDJE at NTCIR-6 CLIR: The Head/Lead Method and Graded Relevance Feedback

At NTCIR-6 CLIR, Toshiba participated in the Monolingual and Bilingual IR tasks covering three topic languages (Japanese, English and Chinese) and one document language (Japanese). For Stage 1 (which is the usual ad hoc task using the new NTCIR6 topics), we submitted two DESCRIPTION runs and two TITLE runs for each topic language. Our first search strategy is Selective Sampling with Memory Resetting, and our second one is the Head/Lead method, which uses the Selective Sampling run as one of the components for data fusion. According to the Relaxed and Rigid Mean Average Precision statistics released by the organisers, we are the top performer in all six subtasks. For Stage 2 (which reused the NTCIR-3, 4 and 5 test collections), we repeated our two Stage 1 strategies in order to enable analysis across all four test collections. Moreover, we conducted some unofficial true relevance feedback experiments by exploiting the graded relevance data provided in the test collections. Our automatic run results show that the Head/Lead method slightly but consistently improves performance, while our unofficial “interactive” run results suggest that graded-relevance metrics favour graded relevance feedback while Average Precision favours binary relevance feedback. In addition, our significance tests suggest that the NTCIR-6 Japanese test collection is “harder” than previous collections.